Transaction Processing over High-speed Networks

By avoiding high cost of disk I/O, memory-resident OLTP databases reduce the runtime of typical singlesited transactions to a fraction of a millisecond. With disk gone from the picture, network has become the next bottleneck. In fact, the traditional wisdom is that network is the new disk, and distributed transactions must be avoided as much as possible, through techniques such as partitioning. However, recent technological trends toward low-latency, high-bandwidth networks which are capable of minimizing communication overhead through Remote Direct Memory Access (RDMA) are changing our view on the fundamental assumption that network is significantly the bottleneck. For example, the latency in InfiniBand FDR/EDR is less than 1μs, two orders of magnitude faster than the traditional 1Gb Ethernet, and is only one order of magnitude slower than local memory access (0.1μs). In this project, we aim to explore the design of a memory-resident OLTP with optimistic concurrency control, in particular snapshot isolation, which leverages RDMA and low latency of fast networks as much as possible. We argue that the new RDMA-enabled architecture is neither shared-memory nor shared-nothing. This new architecture requires us to fundamentally rethink the design of databases. In particular, we plan to address the question that in an RDMA-capable distributed system, what changes have to be made to concurrency control and related components of these databases. We speculate that benefits of using RDMA in OLTP systems are three-fold; first, it decreases the latency of transactions. Second, in this new hybrid architecture, the burden of transaction processing can be potentially shared between servers and clients, resulting in a more balanced design. Finally, separating transaction processing from data management, which are both traditionally handled by the server, could facilitate the scale-out process. Our initial experiments show that exploiting RDMA could result in at least an order of magnitude higher performance compared to the existing design.